Add clusterip exposure for in-cluster-only access - #920

Known gaps, not addressed here:

• loadbalancer → clusterip: the external-IP webhook ignores the address going away, so webhook consumers keep the old LB address (same as loadbalancer → traefik today).
• With grandCentral.exposure unset, a clusterip cluster still gets a public nginx Ingress for grand-central. Cloud-only; a follow-up makes that mapping explicit.
• Resume with clusterip is covered by a unit test only: test-mode get_host needs an LB address, so the e2e tests can't connect to a ClusterIP cluster.

Thanks for explicitly documenting the ClusterIP test-harness limitation above. Our review of commit38d0f3191403d3f6ceb280abc09daae4ced0ca7f confirms the distinction: TESTING=True falls through to the LoadBalancer-address lookup, while normal in-cluster operation uses discovery DNS. We therefore regard this as a known test-coverage limitation, not a production defect. Would you prefer to track support for ClusterIP connectivity in the external e2e harness as a separate follow-up? Our planned EKS verification will run the operator in-cluster and cover ClusterIP connectivity and suspend/resume; those live checks have not run yet.

Yes, a separate follow-up. Production connections don't depend on the exposure: get_host always returns the headless crate-discovery-<name> service, and #920 has a unit test for that. A harness test would mainly guard the test-mode branch. Our e2e suite runs the operator in the pytest process outside the test cluster, so it can only reach CrateDB through a public LB address; a ClusterIP test needs a port-forward or a test-only LB service. Tracked in #923. The full integration suite passed on the current head. Your in-cluster EKS run is the more useful check; please post the results here when you have them.

Following up with the in-cluster EKS results requested above: the scoped ClusterIP checks passed against merged commit 5803af5f8fc9bb475491e47dea20be69be704d9f.

Setup: operator built from that commit using the unchanged upstream Dockerfile, package 2.65.2.dev5+g5803af5f8.d20261023, running in-cluster with TESTING=False; EKS/Kubernetes 1.34.9, linux/amd64; fresh single-node CrateDB 6.2.11, explicit spec.cluster.exposure: clusterip, no externalDNS or grandCentral. JMX exporter 1.2.5, SQL exporter 0.24.10. We reused the existing Helm deployment/RBAC templates from chart 2.63.0 (unchanged in the compared source) with the image/config overrides and the exact CRD from the merged commit, managed through normal GitOps.

Results:

• Fresh creation/bootstrap completed; database health GREEN, database and SQL-exporter containers Ready with zero restarts.
• Client Service was ClusterIP with HTTP/PostgreSQL ports; discovery Service remained headless. Neither had external-DNS/LB annotations, LB ingress or an LB-cleanup finalizer.
• A separate in-cluster client resolved both Service DNS names and executed authenticated SELECT 1 and SELECT count(*) FROM sys.nodes; both returned HTTP 200 and [[1]]. This was genuine in-cluster DNS/SQL, not a port-forward check.
• Updating allowedCIDRs was a no-op as intended: the operator explicitly logged that it has no effect for ClusterIP and skipped the source-range patch; both complete Service specs and identities stayed unchanged. We do not interpret this as network isolation.
• Ordinary suspend/resume (replicas: 1 -> 0 -> 1) preserved both Service UIDs and the client ClusterIP. The database returned GREEN and in-cluster DNS/SQL passed again.
• Separately, while suspended, we deleted only the client Service and resumed. The supported resume path recreated it as ClusterIP, with a new UID/IP as expected, preserved discovery identity, and restored GREEN health and in-cluster DNS/SQL. All non-IP Service spec fields and ownership matched the original. We did not assume Service deletion alone triggers steady-state reconciliation.
• Temporary handler retries for pod termination/database startup resolved successfully. No LoadBalancer-address wait was observed.
• Runtime-account LB inventories before/after remained unchanged; CloudTrail lookup returned no CreateLoadBalancer events for the observed window at collection time. These are bounded observations, not proof of zero transient API attempts, given ingestion delay and lack of continuous audit.
• Disposable workloads, Services, client Jobs and supplied credentials were deleted; both data and retained debug PV/EBS storage were removed. No force deletion or finalizer workaround was needed. The candidate remains Ready under active GitOps.

Limits: this was a disposable DEV verification with no existing production CrateDB consumers, not production/release qualification. We did not test LoadBalancer/Traefik exposure transitions, grandCentral, backup/restore or dedicated-master/multi-node topologies. The Traefik routing API was not installed in this test cluster. We did not run the external pytest-process harness and continue to treat #923 as the separate harness follow-up.

Thanks for the implementation and for clarifying the in-cluster versus external-harness distinction.